Training Feed-forward Neural Networks Using the Gradient Descent Method with the Optimal Stepsize

The most widely used algorithm for training multiplayer feedforward networks, Error BackPropagation (EBP), is an iterative gradient descend algorithm by nature. Variable stepsize is the key to fast convergence of BP networks. A new optimal stepsize algorithm is proposed for accelerating the training process. It modifies the objective function to reduce the computational complexity of the Jacobin and consequently that of Hessian matrices, and hereby directly computes the optimal iterative stepsize. The improved backpropagation algorithm helps alleviating the problem of slow convergence and oscillations. The analysis indicates that the backpropagation with optimal stepsize (BPOS) is more efficient when treating large-scale samples. The numerical experiment results on pattern recognition and function approximation problems show that the proposed algorithm possesses the features of fast convergence and less intensive computational complexity.